1. The Field of the Invention
This invention relates to systems, methods, and computer program products related to translating natural language into user interface actions.
2. Background and Relevant Art
Mobile devices, such as tablet computers and smart phones, have experienced explosive growth and market penetration in recent years. Accompanying this growth, the mobile application market has also experienced tremendous growth. Mobile applications can range from simple games to complex productivity software. In many cases, regardless of the type of mobile application, the mobile application communicates through the Internet to various servers. The servers can be used to store user created content, provide additional features, allow a developer to track software and user behavior, or provide other similar services.
In some cases, mobile applications may act as a web portal to a webpage. Using the mobile application, a user may navigate the webpage's content. In some conventional systems, the mobile application may provide a user experience that is very similar to the user experience provided through a general purpose mobile web browser.
Similar to web pages that are accessed through conventional computer-based web browsers, the content viewable by the mobile application may comprise various pieces of code, sometimes referred to as “tags,” that are configured to execute (i.e., “fire”) when the content is viewed. In at least one conventional form, a tag is a hidden element within a webpage that causes some action to occur on a webpage when it is fired. In particular, in at least one implementation, a tag is a piece of code that a webpage owner can place within a webpage, and allows the webpage owner to track information about the visitors who download and execute the particular piece of code.
For example, a tag can comprise a transparent GIF or a tracking pixel, which is placed within the code of a webpage. The actual transparent GIF or tracking pixel can be stored on a separate server from the actual webpage. The tag is fired when the browser requests the transparent GIF or tracking pixel from the separate server. When the separate server receives the request from the browser, the server can gather various analytic data about the requestor.
Once executed, a tag can gather information such as where the visitor came from (e.g., what webpage or search engine), what keywords were used to find the webpage, whether a particular advertisement directed a visitor to the page, and other related data. In many situations, a webpage can comprise multiple tags from a variety of different parties. For example, a particular webpage may contain a Google Analytics tag, a Facebook tag, a pagecounter tag, a weather tag, and other common tags. In at least one implementation, a tag can merely comprise a particular call to an external advertising server.
As mobile applications have become more pervasive and as more users have begun to access web-based content through custom mobile applications, content providers have also become more interested in ensuring that the various tags embedded within their web content are operating correctly within the mobile applications. In many cases, large amounts of advertising money are dependent upon tags correctly tracking visitors to a website. Keeping proper track of these tags, however, presents several unique challenges.
For example, due to the tremendous variety of mobile applications it is much more difficult to create a “standard” test environment. Additionally, many organizations create a mobile version of their web content, such that audits of their conventional web-based content do not necessarily correlate to audits of their mobile content. Furthermore, some applications have begun to allow use of voice interaction. For instance, some mobile applications allow users to provide certain specified voice commands that, when acted upon, cause the mobile application to perform a given task.
Determining which tasks are intended, however, especially when the user provides verbal input other than a specified command (e.g. a natural language input), is very problematic. Still further, determining which UI elements are to be involved in the action is not a straightforward process.